The goal of Image inpainting is to fill in the missing regions of an image in a sophisticated and plausible way. There are several existing image processing packages already available in R that offers quite useful image editing and processing techniques such as Imager, Magick, and EBImage but they lack proper implementation of Image inpainting algorithms. This package tries to fill that gap by providing R users the ability to utilize a recent C/C++ based library that performs “Non-Local Patch-Based Image Inpainting” on images.
Based on the research of Alasdir et al. [1], a patch based image inpainting is possible that smoothly applies patches and can also match patterns and texture in the nearest areas to the occlusion area. An occlusion area is the the region where image inpainting is applied.
The whole C/C++ library code base was tweaked to comply with Rcpp package’s guidelines. Some of them are to use Rcout instead of cout, Rprintf instead of printf and to use R’s random number generators instead of C/C++’s rand and seed functions, among others. Library’s main method was exposed using [Rcpp::export] attribute and the return values were provided to R code using Rccp’s list data structure.
The algorithm takes two images, one being the actual input image and the second with the occlusion area applied. Background of the second image is blacked out such that only the occlusion area remains white, which also really helps in extracting the occlusion area in the R package. Rest of the arguments control how the algorithm behaves and those include 1. patchSizeX : Size of X patch, which can be between 3L to 13L. 7L is most suitable for images between 512x512 to 800x800 pixels and hence it is the default value 2. patchSizeY : Size of Y patch, which can be between 3L to 13L. 7L is most suitable for images between 512x512 to 800x800 pixels and hence it is the default value 3. nLevels : Number of pyramid levels (0-15). Defaults to automatic selection by the algorithm 4. useFeatures : A boolean value, the algorithm also matches the region’s texture if this is set to true or 1 5. verboseMode : Enables or disables the verbose mode
By using the patch size of 7L by 7L, following image inpainting outputs were obtained [1]
Patch size affect on inpainting
As can be seen that 7x7 is the ideal patch size for regular sized images. The affect of use features argument triggers whether the algorithm should match texture or only apply a solid patch, see image below.
Use of feature or pattern matching
Beside providing an ability to call C/C++ image inpainting function, the motive of this package was to effectively apply and analyze the results using well known R functions such as plot and print, etc. We provide two R classes, inpaint and occlusion. The inpaint class holds a list structure of input images, output images, their paths and, algorithm parameters. Plotting, printing and summarizing this class is enabled by providing custom implementation based on their respective functions.
Whereas, the occlusion class extracts the occlusion regions from the occlusion input image provided by the user and then extracts the same regions from input and output images, to finally plot them.
First call the imageInpaint functions which eventually calls the C/C++ library function in the background
library(imageInpainting)
inputFile = system.file("extdata", "got.png", package = "imageInpainting")
inputOccFile = system.file("extdata", "got_occlusion.png", package = "imageInpainting")
outputFilePath = file.path(getwd(), "got_inpainted.png")
inpaintResult = imageInpaint(inputFile, inputOccFile, outputFilePath)## Reading input image
## xSize : 1214
## ySize : 697
## nChannels : 3
## Reading input occlusion
## xSize : 1214
## ySize : 697
## nChannels : 3
## xSize : 1214, ySize : 697, nTupleSize : 1
## patchSizeX : 7, patchSizeY : 7
## max value : 1.000000, min value : 0.000000
##
##
## *************************
## * INPAINTING PARAMETERS *
## *************************
## Number of levels : 3
## Use features : 1
## Residual threshold: 0.100000
## Maximum number of iterations: 10
## *************************
##
##
## *************************
## * PATCHMATCH PARAMETERS *
## *************************
## Patch size X : 7
## Patch size Y : 7
## Number of propagation/random search iterations: 10
## Random search reduction factor (alpha) : 0.500000
## Maximum search shift allowed (-1 for whole image) : -1.000000
## Full search (should be activated only for experimental purposes !!) : 0
## Verbose mode : 0
##
##
## Features calculation time: 0.536814
## Current pyramid level : 2
##
## Initialisation started
##
##
##
## Initialisation finished
##
##
## Current pyramid level : 1
## Current pyramid level : 0
## Inpainting finished !If we print the inpaintResult, our own S3 implementation of print method is called
##
## Image inpainting result
## Result : success
## Total time taken (sec) : 20.84
## Output image path : /Users/saad/Code/imageInpainting/vignettes/got_inpainted.pngSummary of the inpaintResult is also available in matrix form
## value
## Start time "12:24:25, Aug 04 2019 CEST"
## End time "12:24:46, Aug 04 2019 CEST"
## Time taken (sec) "20.84"
## Result "success"
## Input image path
"/Users/saad/Code/imageInpainting/inst/extdata/got.png"
## Input occlusion image path
"/Users/saad/Code/imageInpainting/inst/extdata/got_occlusion.png"
## Output iamge path
"/Users/saad/Code/imageInpainting/vignettes/got_inpainted.png"
## Patch size X "7"
## Patch size Y "7"
## No. of levels "-1"
## Use features "1"
## Verbose mode "0"To view the inpainted output image alongside input and occlusion images, call the plot function as
Now to extract the occlusion regions, call extractOcclusionRegions method and plot the result as below
There are a few sample images already placed inside inst/extdata directory of the package. Following file names can be used to test them
inputFile = system.file("extdata", "barbara.png", package = "imageInpainting")
inputOccFile = system.file("extdata", "barbara_occlusion.png", package = "imageInpainting")inputFile = system.file("extdata", "car.png", package = "imageInpainting")
inputOccFile = system.file("extdata", "car_occlusion.png", package = "imageInpainting")inputFile = system.file("extdata", "golf.png", package = "imageInpainting")
inputOccFile = system.file("extdata", "golf_occlusion.png", package = "imageInpainting")inputFile = system.file("extdata", "house.png", package = "imageInpainting")
inputOccFile = system.file("extdata", "house_occlusion.png", package = "imageInpainting")After reading input and input occlusion files, call the image inpainting method as explained in the Usage section.